The main source code of cpprb is cpprb/PyReplayBuffer.pyx
. Almost all the public classes and functions are implemented in this file. This file contains more than 2K lines (including blank lines), so that maybe we should divide it. However, if we want to utilize full cdef
functionality, (as far as we know) we have to keep def
in .pyx
and cdef
in .pxd
separately. We feel that makes much harder to maintain and just put everything in the single file.
cpprb/VectorWrapper.pyx
and cpprb/VectorWrapper.pxd
have interface classes between .pyx
and C++. These classes utilize buffer protocol, which enables for us to share large data without copy.
cpprb/ReplayBuffer.hh
contains C++ implementation of PER sampling. In this file, there are a lot of template and SFINAE meta-programming. By using if constexpr
, multi-thread (multi-process) version switches algorithm without runtime cost. In order to support multi-process shared memory, the classes can accept (shared) memory acquired at Python side.
cpprb/SegmentTree.hh
is the one of the oldest code and has segment tree implementation. It can accept shared memory for multi-processing, too. The multi-processing version updates its tree lazily. Internal data structure is described at here.
cpprb/ReplayBuffer.pxd
defines interface classes and functions of cpprb/ReplayBuffer.hh
in order to use them in cpprb/PyReplayBuffer.pyx
.
test/*.py
have unit tests. A lot of regression tests are included, too. Dockerfile
is used for build and test environment. .coveragerc
defines coverage configuration for Coverage.py.
Under site
directory, there are project site source and config file. sphinx
directory contains main page and config file of API documentation. These documentation system is described “Documentation” section below.
benchmark
and benchmark2
directory contain benchmark script and Dockerfile using at Benchmark page. These benchmarks might be outdated and need to be updated.
example
directory has example codes. We need more examples.
CHANGELOG.org
contains change log.
CITATION.cff
is used for citation management at GitHub. Please read GitHub official blog post.
setup.py
is used for package build. MANIFEST.in
specifies package additional files.
.gitlab-ci.yml
defines CI on GitLab CI/CD platform. .github
directory contains CI config files for GitHub Actions. CI settings are described at “CI Configuration” section.
crean_reinstall.sh
is old build script for local development. Recently, we don’t use it and we might remove it in the future.
Build system is defined at setup.py
. We don’t use pyproject.toml
nor requirements.txt
. We know declarative programming is preferable, however, it cannot handle complicated conditional build rules, yet. For example, we would like to use Cython only when .pyx
files exist and they are newer than corresponding .cpp
ones. Source codes hosted at PyPI (not GitLab/GitHub) don’t include .pyx
, then users who use pip command don’t need to prepare Cython at all.
In order to realize single command installation, NumPy and Cython are lazily imported after setup_requires
become available.
Generated markdown of README is re-used as long_description
describing PyPI package page. A unrosolved problem is that we cannot show the cpprb logo on the PyPI page.
Additionally, by setting DEBUG_CPPRB
environment value, we enable debug mode build, where Cython line trace is enabled.
Project site are built by Hugo and hosted at GitLab Pages. The documents are written in Org Mode (site/site.org
) and exported with ox-hugo. (aka. .org
-> .md
-> .html
) This sequence is very useful to reuse (partial) README in the project site. The disadvantage is Org Mode is minor to markdown. site/init.el
defines ox-hugo configuration.
We use Hugo Learn Theme. We also make custom shortcodes at site/layouts/shortcodes
in order to create the highly customized landing page. We still need design improvement to make it easier and more understandable for new commers.
To show star and fork counts on the side bar, we use custom JavaScript defined at site/static/js/custom.js
. We utilize GitLab REST API and GitHub REST API.
API reference is built with Sphinx from docstring at the source codes. The docstring adopts NumPy style, so that Napoleon extension is enabled. By using custom CSS (aka. sphinx/static/custom.css
), we try to match the look and feel with that of the Hugo part. Additionally, we use sphinx-automodapi to build the module structure.
Unit test coverage report are hosted, too. The report is generated by unit test.
We mainly utilize GitLab CI/CD. We execute binary build, unit test, test coverage aggregation, PyPI upload, and project site build and deploy. The configuration is defined at .gitlab-ci.yml
.
Unfortunately, schedule trigger jobs have not been working recently. We have to investigate it.
Additionally, GitHub Actions are used for specific purpose. Since we can use Windows and macOS for free, we execute test script on them and upload Windows binary to PyPI.
We utilize CodeQL for code security and quality analysis. The results are shown here (Is the page public?). To be honest, we are not sure whether this analysis is meaningful for us.